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ABSTRACT 

Separate meta-analyses have been completed recently 
on the effectiveness of computer-based education (CBE) in elementary 
schools, high schools, colleges, and nontraditional postsecondary 
institutions. Several conclusions can bo drawn from these 
meta-analyses. First, although CBA has not been uniformly successful 
in all its guises and at all instructional levels, most CBE programs 
have had the following positive effects on students: students have 
learned more and retained more in courses providing assistance from 
computers; students learned their lessons in less instructional time 
and liked their classes more when they received computer help; and 
students developed positive attitudes towards computers when they 
received help from them in school. Second, results of CBE vary as a 
function of the evaluation design used to measure CBE effectiveness; 
and third, study outcomes also vary as a function of the type of 
publication in which the results are found. Since CBE generally has 
positive effects on student learning, future programs of 
implementation and development should be encouraged. Research is 
needed, however, to determine the factors that produce differences in 
th& results of studies reported in journals and in dissertations, and 
differences in the results of studios using different research 
designs. (Author/JB) 
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Abstract 



Separate meta-analyses have been completed recently on 
the effectiveness of computer-based education (CBE) in 
elementary schools, high schools, colleges, and 
nontraditional postsecondary inst itut ionsc Several general 
conclusions can be drawn from these meta-analyses . First, 
CBE programs usually have positive effects, as measured by 
several different criteria of instructional effectiveness. 
Effects are not uniformly, high, however, for all types of 
CBE programs at all instructional levels. Second, effects 
vary as a function of the evaluation design used to measure 
CBE effectiveness. And third, study outcomes also vary as a 
function of the type of publication in which the results are 
found. 
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Consistencies in Findings on Computer-Based Education 



Many people believe that computer technology will 
transform society in the years ahead as completely as the 
invention of the printing press did 500 years ago or as the 
invention of writing did thousands of years ago. Like these 
earlier inventions, computer technology gives society a new 
way of encoding, storing, and retrieving information. And 
it therefore seems destined to have pervasive and lasting 
effects on our professional and our personal lives. 
Although a few years ago it seemed that future generations 
would remember the 20th century as the atomic age, it now 
appears more likely that they will remember our times as the 
computer age. 

Educational researchers and developers therefore are no 
longer asking whether a computer revolution will occur in 
education. They are asking instead how it will occur. Will 
the changes in education come swiftly and smoothly, or will 
education's transition to the computer age be full of ialse 
starts and costly mistakes? No one asked such questions 
when the revolutions based on writing and printing were 
taking place because they had no good ways of finding 
reliable answers. Researchers are asking these questions 
now because they know that the tools of social science are 
available to help them reach conclusions about computer 
technology's effects. Social scientists have a rare 
opportunity to measure these effects f predict future 
results, and help control developments at a critical point 
in the history of education. 

During the past two decades, a number of researchers 
have used the methods of the social sciences to evaluate 
computer-based teaching. My colleagues and I recently 
joined this effort because we believe that adapting to the 
computer age is one of the major challenges facing schools 
today. But what is the yield from twenty years cf research? 
Do we know anything now that we did not know before? The 
purpose of this paper is to describe and interpret the most 
important findings . 



RESULTS 

We have described the major results at length in other 
places (Bangert-Drowns, J. Kulik, & C. Kulik, in press; 
C. Kulik & J. Kulik, 1985; C. Kulik, J. Kulik, & Shwalb, 
1985; J. Kulik, C. Kulik, & Bangert-Drowns, in press). What 
follows is a summary and integration of results reported 
there. 
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Overall results . Most CBE programs have had positive 
effects on students (Table 1). 

(a) Students have generally learned more in classes when 
they have received help from computers. The average 
effect of computers in all 199 studies used in our 
meta-analyses was to raise examination scores by 0.31 
standard deviations, or from the 50th to the 61st 
percentile. 

(b) Students remembered what they learned longer. The 
average effect of CBE in 18 followup or retention 
studies was to raise examination scores by 0.17 
standard deviations. 

(c) Students also learned their lessons with less 
instructional time. She average reduction in 
instructional time in 28 investigations of this point 
was 32%. 

(d) Students also liked their classes more when they 
received computer help. The average effect of CBE in 
17 studies was to raise att itude-toward-instruct ion 
scores by 0.28 standard deviations. 

(e) Students developed more positive attitudes toward 
computers when they received help from them in school. 
Average ES in 17 studies on attitude toward computers 
was 0.33. 

(f) Computers did not, however, have positive effects in 
every area in which they were studied. The average 
effect of CBE in 29 studies of attitude toward subject 
matter was near zero, and the average effect was also 
near zero in 23 studies of course completion. 

Type of computer use and instructional level * CBE was 
not uniformly successful in all its guises and at all 
instructional levels (Table 2). Effects of computer- 
assisted instruction (CAI ) , computer-managed instruction 
(CMI ) y and computer-enriched instruction (CEI ) were 
different at different instructional levels. 

(a) In elementary schools, for example, CAI programs almost 
always produced good results, raising student 
achievement scores in the typical case by 0.47 standard 
deviations. CMI programs, on the other hand, produced 
much weaker effects, raising student achievement in the 
typical case by only 0.07 standard deviations. No 
adequate evaluations of CEI programs in elementary 
schools turned up in our literature searches. 

(b) In high schools, the pattern of findings was quite 
different. CMI had the most to contribute at this 
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level, and CEI had the least to contribute, CAI 
produced effects that were intermediate in size. 

(c) In colleges and adult settings, CAI , CMI , and CEI were 
all effective. Young adults and older adults seem able 
to profit from a variety of different uses of the 
computer in teaching. 

Study features and study outcomes . A few study 
features were consistently related to outcomes of 
evaluations of computer-based education (Table 3). 

(a) Study results were consistently stronger in published 
studies and weaker in unpublished ones. The average 
effect of CBE in published studies was to raise student 
examination scores by 0.46 standard deviations, whereas 
its average effect in unpublished studies was to raise 
scores by only 0.23 standard deviations. 

(b) Effects were consistently larger in short studies and 
weaker in longer ones. The average effect of CBE in 
short studies was to raise examination scores by 0.36 
standard deviations, whereas the its average effect in 
longer studies was to raise scores by 0.27 standard 
deviations. 

(c) Effects tended to be larger in more recent studies and 
smaller in older studies. The average effect of CBE in 
studies published before 1975 was to raise examination 
scores by 0.24 standard deviations; the average effect 
in studies published in later years was a score 
increase of 0.36 standard deviations. 

(d) Effects tended to be larger when different teachers 
taught the experimental and control groups. Effects 
were smaller when the same teacher was responsible for 
both groups. With the same teacher in charge of 
experimental and control groups, average size of effect 
on examination scores was 0.24 standard deviations. 
With different teachers in charge of the groups, the 
average effect was 0.40 standard deviations. 



DISCUSSION 

Can we conclude from these findings that the computer 
has the potential to add something important to education? 
Some critics have argued from these findings that the 
computer is not the active ingredient in these outcome 
studies. Other factors have been invoked to explain — or 
explain away — the positive findings: editorial gatekeeping, 
novelty factors, and uncontrolled teacher effects. 
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Editorial gatekeeping . Some critics believe that 
results in unpublished reports are more accurate than 
results in journals because unpublished results have not 
been distorted by editorial gatekeeping. This is the 
purported tendency of researchers, reviewers, and editors to 
base publication decisions on size and statistical 
significance of effects rather than on study quality. 

The critics are correct in noting that results differ 
in published and unpublished studies. It should be noted, 
however, that published and unpublished studies differ from 
each other in some other ways too. The authors of journal 
and dissertation studies, for example, differ from one 
another in their research experience, in their resources, in 
their relationship to instructional developers, and in many 
other respects. Such differences can explain — just as well 
as publication bias does — the differences in results found 
in dissertations and journals. It seems to us that we know 
too little about what lies behind the difference in journal 
and dissertation results to reject out-of-hand either kind 
of result. 

Novelty effects . Stated simply, the novelty hypothesis 
says that the novelty of CBE contributes a good deal to its 
effects on students. One prediction from this hypothesis is 
that effects will be greater in short studies and smaller in 
long ones. This is, in fact, the way the results come out. 
Short studies of CBE reported more positive results than did 
long studies: the average ES in short studies was 0.34, and 
the average ES in long studies was 0.26. Critics have 
interpreted this finding to mean that CBE is effective in^ 
raising achievement only while students find it novel. With 
familiarity, CBE loses some of its potency. 

It is important to note that our overall conclusions 
about CBE effectiveness would not be very different if we 
based them on the short-term studies alone or on the long- 
term studies. But it is not obvious to us that one set of 
studies is clearly better than the other. The long-term 
studies provide a better control for novelty effects, but 
the short-term studies may provide a better control over 
other extraneous factors. In short studies, for example, 
criterion tests may measure more exactly the material taught 
by the competing methods. 

It is also important to note that another prediction 
can be made from the novelty hypothesis. And that is that 
CBE will become less effective as computers become more 
common in society. This prediction is not supported by our 
analysis. CBE has become more effective over the years — 
perhaps as a function of improvements in computer software, 
hardware, and courseware. 



ERIC 



7 



Consistencies in Findings - 6 



Experimental design . Critics have also suggested that 
apparent CBE effects may be produced by other uncontrolled 
differences in the treatment of experimental and control 
groups. One indicator that such differences are important, 
these critics believe, is the relationship reported in a 
number of meta-analyses between effect size and the use of 
the same versus different instructors for the experimental 
and control treatments in a study. When the same instructor 
teaches experimental and control treatments, effects are 
often smaller; when different instructors are responsible 
for experimental and conventional teaching, effects are 
often larger. 

Why should one-instructor and two-instructor 
experiments produce somewhat different results? It is not 
at all obvious to us. It may be, for example, that in two- 
instructor experiments, the poorer instructor is usually 
assigned to the control condition and the better instructor 
to the experimental condition, and the difference between 
conditions is magnified because of these teacher 
assignments. If this is the case, then one-instructor 
studies more accurately assess the effects of CBE. It may 
also be, however, that in one-instructor studies there is 
diffusion of the innovative treatment to the control 
condition. Involvement of a teacher in an innovative 
approach to instruction may have a general effect on the 
quality of the instructor's teaching. Outlining objectives, 
constructing lessons, preparing evaluation materials, and 
working with computer materials — requirements in CBE — may 
help a teacher to do a better job in a conventional teaching 
assignment. If this is the case, two-instructor studies 
provide the better basis for estimating the size of an 
experimental effect . 



CONCLUSIONS 

Among the conclusions that can be drawn from CBE 
outcome studies, we believe that the following are 
especially important: 

(a) Most CBE programs have had positive effects on student 
learning. Future programs of implementation and 
development of CBE for schools should therefore be 
encouraged. If such programs are as carefully designed 
as present implementations are, they will most likely 
produce positive results. 

(b) CBE has not been uniformly successful in all its gu'ses 
and at all instructional levels, however. Elementary 
schools have had a good deal of success in increasing 
student achievement through CAI programs; they have had 
less demonstrable success with CMI and CEI . Secondary 
schools have had success with CAI and CMI but less 
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success with CEI. College and adult courses have 
benefited moderately from CAI, CMI, and CEI. Future 
programs of implementation and development should also 
take these findings into account* 

(c) Both journal articles and dissertations present a 
basically positive picture of CBE effectiveness, but the 
findings reported in journal articles are clearly more 
favorable. Reseerch is badly needed to determine the 
factors that produce differences in journal and 
dissertation results because such differences have been 
found in a number of different areas in social science 
research. Does editorial gatekeeping lead professional 
journals to present a distorted picture of social 
science findings? Or do dissertation authors simply 
measure experimental effects less well than do more 
seasoned researchers? 

(d) Although a variety of different research designs can be 
used to show the effectiveness of CBE, certain research 
designs seem to produce more positive results. Studies 
where the same instructor teaches both experimental and 
control classes, for example, report somewhat weaker 
effects than do studies with different experimental and 
control teachers. Studies of lcnq duration often report 
weaker effects than do short studies. Reasons for the 
difference in results from studies using different 
experimental designs are imperfectly understood, 
however. Research on such factors should be encouraged. 
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Table 1 

Average Effect of CBE on Students 
In 199 Studies 



Outcome 


Number 


Average 


Measure 


of Studies 


Effect* 


Pinal examination 


199 


0.31 


Follow-up examination 


18 


0.17 


Attitude toward instruction 


17 


0.28 


Attitude toward computers 


17 


0.23 


Attitude toward subject 


29 


0.05 


Course completion 


23 


-0.06 


Instructional time 


28 


68% 



*The average effect is measured by the statistic ES for 
the following variables; final examination; followup 
examination; attitudes toward computers, instruction, and 
subject matter. Effects were measured by the statistic h 
(Cohen, 1977) for course completion* Savings in 
instructional time were measured in percentage of time 
saved (x/c) . 
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Table 2 



Average 


ES for 


Different 


CBE 


Implementations 




at 


Different Instructional Levels 










Type of 


CBE 


Implementation 




ictructional 
Level 




CAI 




CM I 


CEI 




N 


ES 


N 


ES 


N 


ES 


Elementary 


29 


0.47 


4 


0.07 


0 




Secondary 


17 


0.36 


9 


0.40 


16 


0.07 


College 


58 


0.26 


13 


0.35 


28 


0.23 


Adult 


18 


0.29 


3 


0.72 


2 


1.13 



Note ; CAI is computer-assisted instruction; CMI is 
computer-managed instruction; CEI is computer-enriched 
instruction. 
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Table 3 

Average ES for Final Examinations 
By Study Feature 



Study Feature 

Publication Source 

Published 

Unpublished 
Duration of instruction 

Less than 9 weeks 

9 weeks or more 
Publication year 

Before 1975 

1975 and after 
Control for instructor effect 

Control present 

No control 



Number 
of Studies 

65 
131 

79 
114 

92 
104 

100 
86 



Average ES 

0.46 
0.23 

0.36 
0.27 

0.24 
0.36 

0.24 
0.40 



